SdAE: Self-distillated Masked Autoencoder

With the development of generative-based self-supervised learning (SSL) approaches like BeiT and MAE, how to learn good representations by masking random patches input image reconstructing missing information has grown in concern. However, PeCo need a “pre-pretraining” stage produce discrete codebooks for masked representing. MAE does not require pre-training codebook process, but setting pixels as reconstruction targets may introduce an optimization gap between downstream tasks that quality always lead high descriptive capability model. Considering above issues, this paper, we propose simple Self-distillated AutoEncoder network, namely SdAE. SdAE consists student branch using encoder-decoder structure reconstruct information, teacher producing latent representation tokens. We also analyze build views from perspective bottleneck. After that, multi-fold strategy provide multiple with balanced boosting performance, which can reduce computational complexity. Our approach generalizes well: only 300 epochs pre-training, vanilla ViT-Base model achieves 84.1% fine-tuning accuracy on ImageNet-1 k classification, 48.6 mIOU ADE20K segmentation, 48.9 mAP COCO detection surpasses other methods considerable margin. Code is available at https://github.com/AbrahamYabo/SdAE .